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EXECUTIVE SUMMARY: 
Evaluation of Around the Corner 


Background 

This report highlights findings from an evaluation of Around the Corner (ATC) conducted 
by researchers at the Center for Research and Reform in Education (CRRE) at Johns Hopkins 
University (JHU) and funded by an Investing in Innovation (i3) Fund development grant. The 
Success for All (SFA) Foundation developed Around the Comer (ATC). 

ATC is a program for both preschool and kindergarten students that enables teachers to 
show children how the world works for concepts that are not possible to illustrate in a classroom 
with actual objects. The program includes computer activities and videos for children. 
Opportunities to view videos again at home (Home Links) provide the repeated experiences with 
language concepts and vocabulary that are necessary to language development. An additional 
component of the ATC intervention includes interactive, video-based professional development 
for teachers. 

Program developers at SFA hypothesized that ATC would improve young students’ early 
reading skills. ATC was implemented in the 2014-15 and 2015-16 school years to preschool and 
kindergarten students. This evaluation study estimates the two-year cumulative impact of ATC on 
students’ early reading skills. 

Research Questions 

The research question for the impact study is: What is the cumulative impact over two 
years (preschool to kindergarten) of ATC on students’ early reading skills in comparison with 
children experiencing the business-as-usual condition (e.g., Curiosity Corner and KinderCorner 
without the interactive multimedia aspects)? 

The research questions for the implementation study are: 

1. What is the overall level of fidelity of implementation and how much variation in 
implementation fidelity was there across sites (schools and classrooms)? 

2. What are the attitudes and experiences of teachers and parents regarding the 
intervention? 

3. What are the similarities and differences between ATC and control classes? 

Research Design 

The impact study is a cluster randomized controlled trial in which 12 elementary schools 
already using programs for pre-K and K students created by SFA were randomly assigned to 
either the experimental and control condition (six per condition). All pre-K and K teachers and 
students in those schools participated in the intervention over a two-year period. The study uses 
mixed methods to estimate the program impact and document the fidelity of implementation and 
perceived quality of the program. 
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Sample 

Sample selection targeted SFA’s partner districts, and schools were selected based on 
interest in participating, geographic diversity (e.g., north, south, southwest, and mid-west; rural, 
large, mid-size, and small city), and student need. The impact study uses an intent-to-treat 
student sample. Students were included in the sample if they were in preschool in a study school 
in the 2014-15 school year and in kindergarten in a study school in the 2015-16 school year. 

Measures 

The outcome measures for the impact study are tests that gauge students’ early reading 
skills. The outcome measures were administered to kindergarten students in the spring of the 
2015-16 school year and they are: 

• Woodcock-Johnson III Word Attack subtest (confirmatory) 

• Woodcock-Johnson III Letter-Word Identification subtest (exploratory) 

• Peabody Picture Vocabulary Test (PPVT) (exploratory) 

• Test of Language Development (TOLD-4) Sentence Imitation subtest (exploratory) 

The fidelity of implementation of the ATC program was measured by training of SFA 
coaches, teacher professional development, communication meetings for principals and program 
facilitators, distribution of ATC materials to teachers, and training or other resources provided to 
parents on use of home media. To understand the perceived quality of the program by teachers 
and parents and the degree to which treatment teachers implemented the program in their 
classrooms, additional data were collected: 

• Teacher questionnaires 

• Parent surveys 

• School and classroom implementation ratings 

• School observations and site visit reports 

Analytic Approach 

Multiple linear regression was used to estimate the impact of ATC on students’ early 
reading skills. The model included the treatment indicator, baseline pre-test score, English 
language learner status, baseline performance level, and interaction terms. 

The analysis of qualitative data was guided by Miles and Huberman’s (2004) model, 
consisting of transcribing responses, deriving codes, identifying themes, and revision and 
refinement. Findings were triangulated across multiple data sources per Denzin’s (2001) 
framework. 

Findings 

We found no statistically significant differences between ATC and control students in 
early reading skills, as measured by the confirmatory outcome measure. We also found no 
statistically significant differences between ATC and control students on additional post-tests 
analyzed for exploratory purposes. 
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In the implementation study, we found that the ATC program was implemented with 
fidelity in both implementation years (2014-15 and 2015-16). Further, the data show that ATC 
improved home-to-school interactions and student engagement according to teachers and 
parents. The multimedia component in ATC afforded parents and teachers overlapping 
understandings of what children were learning. Parents reported that the home videos increased 
their child’s engagement in school and reinforced what their child learned at school. Finally, the 
vast majority of teachers believed that ATC benefitted students’ language and literacy skills. 

Conclusion 

In evaluating the two-year cumulative impact of the ATC program on young children’s 
early reading skills, we found no statistically significant differences in early reading skills for 
ATC and control students on the confirmatory outcome measure. The findings of the impact 
study suggest that the evaluation study may have been underpowered and that the effect of ATC 
on some measures of early reading skills compared with the business-as-usual condition may be 
directionally positive on average but less than 0.20 standard deviations. More research is needed 
with larger sample sizes and lower minimally detectable effect sizes. 

Another limitation of this study is that the business-as-usual condition consists of 
programs that are, at least in some ways, similar to the ATC intervention. Thus, the effect size of 
ATC may be much greater in a context where ATC participation is compared with a lower quality 
preschool or kindergarten program. 

It is also noteworthy that qualitative data revealed positive attitudes by both teachers and 
parents regarding the impacts of ATC on student engagement and improving school-to-home 
connections. If proven to be replicable and generalizable in future studies, the overall results of 
this study, therefore, have educational significance by showing ATC to produce comparable 
achievement results to its already research-supported core programs (Curiosity Corner and 
KinderComer) while potentially increasing student engagement and parental involvement in their 
children’s education. 
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Evaluation of Around the Corner 

This report highlights findings from an evaluation of Around the Corner (ATC) conducted 
by researchers at the Center for Research and Reform in Education (CRRE) at Johns Hopkins 
University (JHU) and funded by an Investing in Innovation (i3) Fund development grant. The 
Success for All (SFA) Foundation developed ATC for preschool and kindergarten students. 
Program developers hypothesized that ATC would improve young students’ early reading skills. 
ATC was implemented in the 2014-15 and 2015-16 school years to preschool and kindergarten 
students. This evaluation study estimates the two-year cumulative impact of ATC on students’ 
early reading skills. 


Background 

Curiosity Corner is a current comprehensive program offered by the SFA Foundation for 
three- and four-year-old preschoolers. The program is designed to provide a strong foundation in 
language and literacy, mathematics, science, listening and social skills, creative expression, and 
positive self-esteem through a holistic, thematic approach to instruction. KinderComer is a 
current comprehensive kindergarten program offered by SFA based on research indicating that 
young children learn best when material is delivered holistically rather than in isolation. 

The main focus of Curiosity Corner is on language development; children are exposed to 
a broad range of language-rich experiences focusing on themes that build out from the children 
themselves, their families, and their neighborhoods, to age-appropriate experiences with the 
languages of science, social studies, mathematics, and other areas. KinderComer shares with 
Curiosity Corner a focus on language development, cooperative learning, exploration, phonemic 
awareness, and development of the whole child. It adds a focus on phonics, vocabulary, and 
comprehension, using fast-paced instruction, paired reading activities, and comprehension¬ 
building experiences. 

Around the Comer (ATC) is a program for both preschool and kindergarten students that 
combines and extends components of both Curiosity Comer and KinderComer. Technology 
enhancements around embedded multimedia enable teachers to show children how the world 
works for concepts that are not possible to illustrate in a classroom with actual objects. The 
program includes computer activities and videos for children. Opportunities to view videos again 
at home (Home Links), with or without family members, provide the repeated experiences with 
language concepts and vocabulary that are necessary to language development. An additional 
component of the ATC intervention includes interactive, video-based professional development 
for teachers. 

ATC is based on the rationale and research evidence showing that video content greatly 
enhances children’s learning if it directly reinforces instructional objectives. For example, 
research on the PBS television show Between the Lions , which uses puppets, animation, and live 
action to build early reading skills, has shown significant positive effects of this program for 
young learners (Linebarger, 2000; Linebarger, Kosanic, Greenwood, & Doku, 2004). Also, thirty 
years of research shows positive impacts of Sesame Street (Fisch & Truglio, 2000; Rice, Huston, 
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Truglio, & Wright, 1990). ATC uses this impactful content in a more intentional manner, to 
directly reinforce teachers’ lessons. 

Embedded multimedia refers to the use of brief visual content, such as video, pictures and 
linked text, into daily lessons. By directly linking visual and auditory instruction, embedded 
multimedia can enhance students’ cognition and motivation (Mayer, 2008). Research finds that 
animations, with content closely linked to the content being taught, can greatly increase learning 
(Hoeffler & Leutner, 2006; Neuman, 2006; Savage et al., 2010). 

Research Questions 

The research question for the impact study is: What is the cumulative impact over two 
years (preschool to kindergarten) of ATC on students’ early reading skills in comparison with 
children experiencing the business-as-usual condition (e.g., Curiosity Corner and KinderCorner 
without the interactive multimedia aspects)? 

The research questions for the implementation study are: 

1. What is the overall level of fidelity of implementation and how much variation in 
implementation fidelity was there across sites (schools and classrooms)? 

2. What are the attitudes and experiences of teachers and parents regarding the 
intervention? 

3. What are the similarities and differences between ATC and control classes? 

Method 


Research Design 

The impact study is a cluster randomized controlled trial in which 12 elementary schools 
already using SFA’s Curiosity Corner and KinderCorner in pre-K and K were randomly assigned 
to either the experimental and control condition (six per condition). The pre-K and K teachers in 
these schools will teach according to their assigned treatment (ATC or the existing SFA 
programs), and all pre-K and K students in ATC schools will receive the intervention over a two- 
year period. The study uses mixed methods to estimate the program impact and document the 
fidelity of implementation and perceived quality of the program. 

Sample 

Sample selection targeted the partner districts that were already implementing Curiosity 
Corner and KinderCorner. Districts signed letters of intent to participate in January, 2014. From 
these districts, 12 schools were selected based on interest in participating, geographic diversity 
(e.g., north, south, southwest, and mid-west; rural, large, mid-size, and small city), and student 
need. Randomization occurred on March 10, 2014. ATC is designed to enhance pre-reading and 
language skills of young children in virtually any preschool and kindergarten context; however, it 
is primarily targeted toward disadvantaged and minority children who enter preschool far behind 
middle-class students in exposure to pre-reading skills and vocabulary. The program also 
strongly focuses on students who are English language learners (ELLs). It meets the needs of 
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ELLs by modifying instruction, providing constant opportunities to use English generatively, and 
using video and pictures to build English vocabulary. 

District research agreements were effective July 1, 2014, providing consent at the school 
level for participation in the evaluation. Student-level consent was not obtained as the study was 
deemed exempt under IRB regulations based on the following: This was a school-level random 
assignment that utilized standard educational testing to evaluate instructional practices. Results 
were reported in the aggregate with no identifying information linked to any individual 
participant. However, oral assent was obtained because students were 3-5 years old and not able 
to read or fully understand written consent. Students were included in the sample if they were in 
preschool in a study school in the 2014-15 school year and in kindergarten in a study school in 
the 2015-16 school year. Students were included in the sample if they had non-missing pre- and 
post-test scores. 1 

Table 1 

District Characteristics 


District name 


Geographic location 

Urbanicity 2 

Greater Johnstown, Pennsylvania 
St. Martin Parish, Louisiana 
Morgan County, Missouri 
Alhambra, Arizona 

Steubenville City, Ohio 


Mid-Atlantic 

South 

South 

Southwest 

Midwest 

Midsize city 

Rural 

Rural 

Large city 

Small city 

Table 2 




Student Characteristics 




Ah students 

Treatment 

Control 

Black 

16.8% 

26.4% 

7.0% 

White 

23.6% 

17.4% 

29.9% 

Latino 

42.8% 

38.0% 

47.6% 

Other Race/Ethnicity 

16.8% 

18.1% 

15.5% 

FARMS 

99.5% 

100.0% 

98.9% 


1 No missing values on either the pre- or post-test were imputed. 

2 Urbanicity was collected from the National Center for Education Statistics’ district directory information for the 
2013-14 school year. 
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All students 

Treatment 

Control 

ELL 

30.2% 

26.7% 

33.8% 

Special education 

7.5% 

7.9% 

7.0% 

Female 

48.5% 

48.4% 

48.7% 
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Measures and Instruments 

The pre- and post-tests for the impact study are tests that gauge students’ early reading 
skills. Baseline testing occurred between September - November, 2014, and all students enrolled 
in the participating schools at the time were assessed. There were no schools or students who 
opted out of the assessment. Pretests were administered to verify the comparability at baseline of 
the experimental and control student samples and to serve as a covariate in the model to estimate 
program impacts. Preschool students were pretested in the fall of the 2014-15 school year using 
the Peabody Picture Vocabulary Test (PPVT-III) (Dunn, & Dunn, 1997), a standardized, 
individually-administered measure of children’s receptive vocabulary. The post-tests were 
administered to kindergarten students in the spring of the 2015-16 school year during a three to 
four week testing window. The post-tests analyzed were: 

• Woodcock-Johnson III Word Attack subtest (confirmatory) 

• Woodcock-Johnson III Letter-Word Identification subtest (exploratory) 

• Peabody Picture Vocabulary Test (exploratory) 

• Test of Language Development (TOLD-4) Sentence Imitation subtest (exploratory) 

The fidelity of implementation of the ATC program was detennined on the basis of 
whether coaches attended trainings, treatment teachers attended professional development, 
coaches conducted site visits to schools and communicated with principals, treatment teachers 
received the necessary ATC lesson materials and technological equipment, and training or other 
resources were provided to parents on use of home media. 

To understand the perceived quality of the program by teachers and parents and the 
degree to which treatment teachers implemented the program in their classrooms, additional data 
were collected: 

• Teacher questionnaires were administered in spring 2015 and fall 2015 to treatment 
teachers. The questionnaires gauged teachers’ perception of the ATC program, as well 
as its benefits to the students. 3 

• Parent surveys were administered in spring 2015 and fall 2015 to parents of treatment 
students. The surveys gauged parent feedback regarding children’s use of and 
engagement in interactive videos. 4 

• Snapshots and school visit reports were conducted multiple times each year in SYs 
2014-15 and 2015-16 in both treatment and control schools. Snapshots are 
observations of school-wide phenomena (e.g., student engagement and instructional 
processes). School visit reports are summaries of meetings between program coaches 
and school staff that document school accomplishments and further goals in 
implementing the SFA programs (Curiosity Corner and KinderComer for control 
schools, and the programs with the multimedia component for treatment schools). 

• School ratings were determined once each year for SYs 2014-15 and 2015-16 for 
treatment schools. These ratings reflected classroom and school fidelity of program 
implementation, as well as instructor efficacy. 


3 For the spring 2015 teacher questionnaire, 100% of the 19 treatment teachers responded. For the fall 2015 teacher 
questionnaire, 24 of 44 treatment responded, a return rate of 55%. Results for the fall 2015 and spring 2015 teacher 
questionnaires were similar. 

4 The parent respondent rate was 63% in fall 2015 and 52% in spring 2015. 
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All data for the implementation study were collected by SFA. 

Analytic Approach 

Multiple linear regression was used to estimate the impact of ATC on students’ early 
reading skills. The following model was used to estimate program impacts: 

Y = Po + Pi(Pretest) + P 2 (ATC) + P3(ELL) + P4(ATC*ELL) + P5(Baseline Performance 
Level) + p 6 (ATC*Baseline Performance Level) + s 

Where Y is the outcome of interest, Po is the constant; Pretest is the student’s baseline PPVT 
score; ATC is an indicator for whether a student was attending a school participating in ATC; 
ELL denotes whether the student was an English language learner; Baseline Performance Level 
was whether the student scored low, average, or high on the pretest 5 ; ATC*ELL is the interaction 
tenn between ATC and ELL; ATC*Bascline Perfonnance Level is the interaction tenn between 
ATC and Baseline Performance Level; and £ is the residual. Our primary interest is on the 
coefficient p 2 , which estimates the intent-to-treat impact of ATC on students’ early reading skills. 
To account for the multilevel structure of the data, the standard errors were adjusted for 
clustering at the school level. 6 

We analyzed the qualitative data for the implementation study using Miles and 
Hubennan’s (2004) model, which consists of transcribing responses, deriving codes, identifying 
themes, and revision and refinement. Findings were then triangulated across multiple data 
sources per Denzin’s (2001) framework. Qualitative data include teacher questionnaires, parent 
surveys, snapshots, and school visit reports. School ratings and parent surveys were analyzed 
using descriptive statistics. 


Findings 


Impact Study 

We found no statistically significant differences for ATC and control students in early 
reading skills, as measured by the confirmatory measure, the Word Attack subtest. The estimated 
standardized effect size indicates that ATC may have positively impacted students’ early reading 
skills, however, the estimated standardized effect size (0.10, p=.62) was smaller than anticipated. 
This evaluation study was powered to detect an effect size of 0.20 standard deviations or greater. 7 


5 Students were divided into three equal groups based on their performance on the pre-test: low, average, or high. 
Baseline performance level was coded as 1 (low), 2 (average), or 3 (high). However, the impact estimate was 
obtained using a pairwise comparison using analysis of covariance. The impact estimate represents the adjusted 
mean difference with respect to the regression model between treatment and control students for students in each 
baseline performance level subgroup. 

6 The standard errors were adjusted using a robust cluster standard error option in SPSS. 

7 The Pearson correlation between the pre and post-test was .12. 
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Table 3 

Regression Results for Main Confirmatory Contrast 


Sample School Student Impact Comparison Standardized P- 
N N estimate group mean effect size value 

Outcome (Standard 

variable error) 

Woodcock- All students 6 548 T50 460.54 (UK) flT 

Johnson III (confirmatory) (5.07) 

Word Attack _ 

Table notes: 

(a) Impact and cluster and student attrition tables are included in the Appendices. 

(b) The standardized effect size is the treatment effect estimated by the multiple linear regression model 
divided by the pooled unadjusted standard deviations in the treatment and control groups. 

(c) The standard errors and p-values were adjusted for clustering at the school level. 

The estimated impacts of ATC were also analyzed for exploratory purposes using a 
number of additional post-tests. These findings also showed no statistically significant 
differences for ATC and control students. Regression results from additional post-tests are not 
provided in the previous table nor in the Appendix because the outcome measures were analyzed 
for exploratory purposes only. Standardized effect sizes for all post-tests are shown in Figure 1 
below. 

Figure 1 

Standardized Effect Sizes of Impact of ATC on Students ’Early Reading Skills 



Woodcock Johnson Word Attack 0.10 Woodcock Johnson Letter-Word Identification -0.01 
Peabody Picture Vocabulary Test -0.14 Test of Language Development Sentence Imitation -0.10 
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Implementation Study 

The qualitative data show that ATC improved home-to-school interactions, as well as 
student engagement. The introduction of Home Links (HL) created content interaction and 
reinforcement between school and home while affording parents and teachers overlapping 
understanding of what children were learning. One teacher noted that “vocabulary is spilling 
over and being used” in the classroom and at home. Based on their conversations with parents, 
teachers also described how parents, in watching the HL DVDs with their child, felt more 
connected to what was happening in class. 

Many parents also commented that the HL DVDs provided them with a better 
understanding of what their child was doing in school because their children watched the DVDs 
at home. As one parent noted, “It allows me to see what my daughter is learning each day in 
school and engage her in more conversation about it.” Many parents stated that the DVDs 
reinforced what the child learned at school, and many parents also noticed an increase in their 
child’s engagement in schoolwork and homework due to the HL DVDs. In general, according to 
parents, a majority of children watched HL DVDs every day; greatly enjoyed watching the 
videos; watched with an adult and/or a sibling; and enjoyed the accompanying stories (see Figure 
2 ). 
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Figure 2 

Parent Survey Results 


100 

90 

80 

70 

60 


3 ? 50 



Note: This figure presents the spring 2015 parent survey results. The parent respondent rate was 52%. 

Results: Child watches every day 68%; 1-3 times per week 29%; never watches 2% 

Greatly enjoys DVDs 84%; Somewhat enjoys DVDs 11%; Doesn’t enjoy DVDs 5% 

Child watches with adult 93%; watches with sibling 89%; watches alone 7% 

Child enjoys stories 60%; Dislikes stories 39% 

The vast majority (93%) of teachers also believed that ATC benefitted students’ language 
and literacy skills. Forty-eight percent believed that ATC benefitted students’ development “a 
great deal,” and the other 52% believed that ATC benefitted students’ development only 
“somewhat” or “not much.” Teachers noted that the most valuable aspects of ATC content were 
the songs, music, and dancing, as well as the familiarity (for students) of the Sesame Street 
characters. Twenty-nine percent of teachers considered the stories the least valuable component; 
some teachers recommended that the videos be shortened, as some videos were considered too 
long and student attention waned. Finally, several teachers suggested that the videos be made 
available on other platforms such as YouTube or made smart phone accessible. 

According to parents, their children particularly enjoyed the interactive singing and 
dancing, and the alphabet, letters, and number sections of the videos. Approximately 40% of 
parents also noted that the stories were too long or caused the child to become distracted, 
although 60% of parents stated that their child liked the stories. Several parents suggested more 
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math, writing, and spelling activities. Other parents would also like the introduction of activities 
which did not necessarily revolve around watching television. 

Regarding fidelity of ATC implementation, we found that in both implementation school 
years (2014-15 and 2015-16), the program was implemented with fidelity in four out of five 
program components: training of SFA coaches, teacher professional development, 
communication meetings for principals and program facilitators, and distribution of ATC 
materials to teachers. We did not have evidence to determine whether the fifth program 
component — training or other resources for parents on use of home media — was implemented. 

We found little variation in the fidelity of ATC implementation across sites (schools and 
classrooms). However, there was some variation in how SFA coaches rated treatment schools and 
classrooms on the basis of fidelity of program implementation and instructor efficacy (see Figure 
3). 

Figure 3 

SFA School Ratings for Program Implementation and Instructor Efficacy in Treatment Schools 


Excellent 4 



Breaux Carol Peck Catalina Cordova East Rll South West Side Westwood 
Bridge Ventura 


■ 2014-15 ■ 2015-16 

Table Notes: 

(a) SFA school ratings were calculated by averaging teacher ratings for teachers in the same school. 

(b) Due to redistricting, preschool students were relocated to two additional schools - Peck and Catalina Ventura - 
for their kindergarten year. ATC was implemented in these schools in 2015-16. 

For 2014-15 the ratings ranged from 2.25 to 3.5; for 2015-16 the ratings ranged from 2.21 to 3.7. 

Finally, snapshots and school visit reports were analyzed to understand differences in 
treatment and control classrooms. Snapshots and school visit reports demonstrated that treatment 
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schools made greater progress in their goals than did control schools. Although control schools 
experienced similar successes and challenges in program implementation as treatment schools, 
observers noted that treatment schools made greater progress in building on their successes and 
addressing their challenges than control schools. Observers noted that in treatment schools, 
students had become familiar with the routine of the ATC program; there was more cooperation 
and teamwork among students; teachers were providing positive reinforcement; and students 
seemed engaged in the content. 

Conclusion 

In evaluating the two-year cumulative impact of the ATC program on young children’s 
early reading skills, we found no statistically significant differences in early reading skills for 
ATC and control students on the confirmatory outcome measure. The findings of the impact 
study suggest that the evaluation study may have been underpowered and that the effect of ATC 
on some measures of early reading skills compared with the business-as-usual condition may be 
directionally positive on average but less than 0.20 standard deviations. More research is needed 
with larger sample sizes and lower minimally detectable effect sizes. 

Another limitation of this study is that the business-as-usual condition consists of 
programs that are, at least in some ways, similar to the ATC intervention. Thus, the effect size of 
ATC may be much greater in a context where ATC participation is compared with a lower quality 
preschool or kindergarten program. 

In the implementation study, we found that the ATC program was implemented with 
fidelity in both implementation years (2014-15 and 2015-16). Further, the data show that ATC 
improved home-to-school interactions and student engagement according to teachers and 
parents. The multimedia component in ATC afforded parents and teachers overlapping 
understandings of what children were learning. Parents reported that the home videos increased 
their child’s engagement in school and reinforced what their child learned at school. The vast 
majority of teachers also believed that ATC benefitted students’ language and literacy skills, 
although teachers held mixed opinions about to what extent. If proven to be replicable and 
generalizable in future studies, the overall results of this study, therefore, have educational 
significance by showing ATC to produce comparable achievement results to its already research- 
supported core programs (Curiosity Corner and KinderCorner) while potentially increasing 
student engagement and parental involvement in their children’s education. 
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Appendix 

This technical appendix includes the following tables: 

• Table 2: Impact Table 

• Table 3.A: Base Sample Sizes of Clusters 

• Table 3.B: Base Sample Sizes of Students 

• Table 4.A: Base Equivalence of Students 

• Table 4.B: Base Equivalence of Clusters 

• Table 5.1: Key Components of Intervention 

• Table 5.2: Fidelity of Implementation of Intervention by Year 
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Table 2: Impact Table 


A 

B 

C 

D 

E 

F 

G 

H 

1 

J 

K 

L 

M 

N 

0 

P 

Q 

Contra 
st Id# 

Contrast 

Name 

(Optional) 

Post-test 

Measure 

Name 

Treatme 
nt Group 
Not 
Clusters 

Treatme 
nt Group 
Not 

Students 

Comparis 
on Group 
Not 
Clusters 

Comparis 
on Group 
Not 

Students 

Unadjust 

ed 

Treatmen 
t Group 

SD 

Unadjuste 

d 

Comparis 
on Group 
SD 

Standar 

d 

Deviatio 

n 

Source 

(Code) 

Comparis 
on Group 
Mean 
(Optional) 

Impact 

Estimat 

e 

Standardiz 
ed Effect 
Size 

(Optional) 

Impact 
Standar 
d Error 

P- 

value 

Code for 
Impact 
Model 
Descripti 
on 

Degree 
s of 
Freedo 

m 

C-WA- 

Conti rmator 
y-Word 

Woodcoc 

k- 












0.62 

2 



F-Y2 

Attack-Full 
Sample- 
Year 2 

Johnson 

III Word 
Attack 

6 

276 

6 

272 

25.79 

24.59 

A 

460.54 

2.50 

0.10 

5.07 

B 

1 


Table Notes: 

(a) Model B is Y = (3o + pi (Pretest) + ^(ATC) + (UELL) + P4(ATC*ELL) + (^(Baseline Performance Level) + (36(ATC‘Baseline Performance Level) + z. 

(b) The impact estimate is the adjusted mean difference (T-C). 

(c) The standard errors and p-vaiues were adjusted for clustering at the school level using a robust standard error estimate in SPSS. 

(d) No adjustments were made for multiple comparisons given the single confirmatory contrast. 
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Table 3.A: Base Sample Sizes of Clusters (Schools) 




Contrast 


C-WA-F-Y2 


Confirmatory-Word Attack-Full Sample-Year 2 

Row# 

T 

C 

1 

# of schools Randomized 

6 

6 

2 

# of schools in Impact Analysis 

6 

6 

3 

Difference (Row1-Row2) 

[if difference = 0, skip to row 8 and enter “0” again. No need to indicate Reason for Loss] 

0 

0 

4 

Joiners included (Yes or No) 

No 

No 

Reason for Loss (Evaluator lists reason for loss - add as many rows as needed 3 

5 




6 




7 




8 

Total Loss (Sum of “reasons for loss” rows - 

This number should equal “Difference” shown in Row 3) 

0 

0 

a Most Reasons for Loss will be considered endogenous; if evaluator considers a reason for loss to be exogenous, enter sufficient information to explain why 
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Table 3.B: Base Sample Sizes of Students 




Contrast 

C-WA-F-Y2 

Confirmatory-Word Attack-Full Sample-Year 2 

Row# 

T 

C 

1 

# of students randomized 3 

435 

390 

2 

# of students in impact analysis 3 

276 

272 

3 

Difference (Row1-Row2) 

[if difference = 0, skip to row 8 and enter “0” again. No need to indicate Reason for Loss] 

159 

118 

4 

Joiners included (Yes or No) 

No 

No 

Reason for Loss (Evaluator lists reason for loss - add as many rows as needed 6 

5 




6 




7 




8 

Total Loss (Sum of “reasons for loss” rows - 

This number should equal “Difference” shown in Row 3) 

159 

118 

3 In studies with randomization of clusters and no joiners, the numbers in this table should reflect only the counts of students in non-attrited clusters. 
b Most Reasons for Loss will be considered endogenous; if evaluator considers a reason for loss to be exogenous, enter sufficient information to explain why 
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Table 4.A: Baseline Equivalence of Students 


A 

B 

c 

D 

E 

F 

G 

H 

1 

J 

K 

L 

M 

Contrast 

ID# 

Contrast 

Name 

(Optional) 

Pre-test 

Measure 

Name 

Treatment 
Group N 

Comparison 
Group N 

Unadjusted 
Treatment 
Group SD 

Unadjusted 
Comparison 
Group SD 

Standard 

Deviation 

Source 

(Code) 

Comparison 
Group Mean 
(Optional) 

Treatment - 
Comparison 
Difference 

Standardized 
T-C Difference 
(Optional) 

Pre-test 
shown in 
this row 
was used 

as a 

control in 
the 

impact 
model for 
this 

contrast 
? (Y/N) 

Code forT- 
C Difference 
Calculation 

C-WA-F- 

Y2 

Confirmatory- 
Word Attack- 
Full Sample- 
Year 2 

PPVT-III 

276 

272 

17.56 

19.05 

A 

89.27 

1.27 

0.07 

Y 

B 


Table 4.B: Baseline Equivalence of Clusters 


A 

B 

c 

D 

E 

LL_ 

G 

H 

1 

J 

K 

L 

M 

Contrast 

ID# 

Contrast Name 
(Optional) 

Pre-test 

Measure 

Name 

Treatment 
Group N 

Comparison 
Group N 

Unadjusted 
Treatment 
Group SD 

Unadjusted 
Comparison 
Group SD 

Standard 

Deviation 

Source 

(Code) 

Comparison 
Group Mean 
(Optional) 

Treatment - 
Comparison 
Difference 

Standardized 
T-C Difference 
(Optional) 

Pre-test 
shown in 
this row was 
used as a 
control in 
the impact 
model for 
this contrast 
? (Y/N) 

Code for T-C 
Difference 
Calculation 

C-WA- 

F-Y2 

Confirmatory- 
Word Attack- 
Full Sample- 
Year 2 

PPVT-III 

6 

6 

9.54 

12.82 

D 

92.08 

0.22 

0.02 

Y 

B 
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Table 5.1: Description of Key Components 


Planned Intervention Activity: 

All key components measured across years of implementation 

List of Key Indicators For Each Key Component 

Train SFA Coaches on new media 

Coach receives 4 hours of training on use of enhanced media components in each implementation year. 

Teacher Professional Development 

Pre-K teachers receive 6-12 hours of initial professional development and a minimum of three classroom 
observation and feedback visits in 2014-15. 

K teachers receive 1-4 hours of PD and 2-4 classroom observation and feedback visits in 2015-16. 

Communication meetings for principals and facilitators 

Principals receive 3-4 communication visits in 2014-15. 

Principals receive 2-4 communication visits in 2015-16. 

Distribution of ATC Materials 

Each Pre-K (2014-15) and K (2015-16) teacher receives a full set of lesson materials and technology equipment 
in each implementation year. 
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Table 5.2: Fidelity of Implementation of Intervention by Year 


Findings from Evaluator Study of Implementation: IMPLEMENTATION YEAR 1 

Enter calendar year: 2014-2015 School Year 

Intervention 
Components: 
Copy from list 
above 

Implementation 
measure (total number 
of measurable 
indicators representing 
each component 

Sample Size at 
the Sample 
Level (# of 
schools, 
districts, etc) 

Representativeness of sample: 
Measured on All (A), Some (S), 
or None (N) of the units 
representing the intervention 
group in the impact analyses 6 

Component Level Threshold 
for Fidelity of Implementation 
for the Unit that is the Basis 
for the Sample-Level 

Evaluator’s Criteria 
for “Implemented 
with Fidelity” at 
Sample Level 

Component 
Level Fidelity 
Score for the 
Entire Sample 

Implemented 
with Fidelity? 

(Yes, No, N/A) 

Planned Intervention Activities [i.e., key components] 

Train SFA Coaches 
on new media 

1 Indicator 

All coaches 
within 6 

Schools 

A 

High participation at coach- 
level = 4 hours of training on 
use of enhanced media 
components 

High participation at school- 
level = 80-100% of coaches 
with high participation 

At least 90% of 
schools with high 
participation 

100% 

Yes 

Teacher 

Professional 

Development 

1 Indicator (Note: 
definition differs 
between Year 1 and 
Year 2) 

All pre-k 
teachers 
within 6 

Schools 

A 

High participation at teacher- 
level = 6-12 hours of initial 
professional development 
and a minimum of three 
classroom observation and 
feedback visits 

High participation at the 
school-level = 80-100% of 
teachers with high 
participation 

At least 90% of 
schools with high 
participation 

100% 

Yes 

Communication 
meetings for 
principals and 
facilitators 

1 Indicator 
(Note: definition 
differs between Year 1 
and Year 2) 

Principals 
within 6 pre-k 
Schools 

A 

High participation at the 
school-level = High 
participation at the principal- 
level = Principal receives 3-4 
“communication visits” 

At least 90% of 
schools with high 
participation 

100% 

Yes 

Distribution of ATC 
Materials 

1 Indicator 

Pre-K 
teachers 
within 6 

Schools 

A 

High participation at the 
teacher-level = receives a full 
set of lesson materials and 
technology equipment 

At least 90% of 
schools with high 
participation 

100% 

Yes 
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Findings from Evaluator Study of Implementation: IMPLEMENTATION YEAR 1 

Enter calendar year: 2014-2015 School Year 

Intervention 
Components: 
Copy from list 
above 

Implementation 
measure (total number 
of measurable 
indicators representing 
each component 

Sample Size at 
the Sample 
Level (# of 
schools, 
districts, etc) 

Representativeness of sample: 
Measured on All (A), Some (S), 
or None (N) of the units 
representing the intervention 
group in the impact analyses b 

Component Level Threshold 
for Fidelity of Implementation 
for the Unit that is the Basis 
for the Sample-Level 

Evaluator’s Criteria 
for “Implemented 
with Fidelity” at 
Sample Level 

Component 
Level Fidelity 
Score for the 
Entire Sample 

Implemented 
with Fidelity? 

(Yes, No, N/A) 





High participation at the 
school-level = 

80-100% of teachers with 
high participation 
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Table 5.2 (Continued): Fidelity of Implementation of Intervention by Year 


Findings from Evaluator Study of Implementation: IMPLEMENTATION YEAR 2 

Enter calendar year: 2015-2016 School Year 

Intervention 
Components: 
Copy from list 
above 

Implementation 
measure (total number 
of measurable 
indicators representing 
each component 

Sample Size at 
the Sample 
Level (# of 
schools, 
districts, etc) 

Representativeness of sample: 
Measured on All (A), Some (S), 
or None (N) of the units 
representing the intervention 
group in the impact analyses 6 

Component Level Threshold 
for Fidelity of Implementation 
for the Unit that is the Basis 
for the Sample-Level 

Evaluator’s Criteria 
for “Implemented 
with Fidelity” at 
Sample Level 

Component 
Level Fidelity 
Score for the 
Entire Sample 

Implemented 
with Fidelity? 

(Yes, No, N/A) 

Planned Intervention Activities [i.e., key components] 

Train SFA Coaches 
on new media 

1 Indicator 

All coaches 
within 6 

Schools 

A 

High participation at coach- 
level = 4 hours of training on 
use of enhanced media 
components 

High participation at school- 
level = 80-100% of coaches 
with high participation 

At least 90% of 
schools with high 
participation 

100% 

Yes 

Teacher 

Professional 

Development 

1 Indicator (Note: 
definition differs 
between Year 1 and 
Year 2) 

All K teachers 
within 6 

Schools 

A 

High participation at teacher- 
level = 1-3 hours of initial 
professional development 
and a minimum of three 
classroom observation and 
feedback visits 

High participation at the 
school-level = 80-100% of 
teachers with high 
participation 

At least 90% of 
schools with high 
participation 

100% 

Yes 

Communication 
meetings for 
principals and 
facilitators 

1 Indicator 

Principals 
within 6 

Schools 

A 

High participation at the 
school-level = High 
participation at the principal- 
level = Principal receives 2-4 
“communication visits” 

At least 90% of 
schools with high 
participation 

100% 

Yes 

Distribution of ATC 
Materials 

1 Indicator 

K teachers 
within 6 

Schools 

A 

High participation at the 
teacher-level = receives a 
full set of lesson materials 
and technology equipment 

At least 90% of 
schools with high 
participation 

100% 

Yes 
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Findings from Evaluator Study of Implementation: IMPLEMENTATION YEAR 2 

Enter calendar year: 2015-2016 School Year 

Intervention 
Components: 
Copy from list 
above 

Implementation 
measure (total number 
of measurable 
indicators representing 
each component 

Sample Size at 
the Sample 
Level (# of 
schools, 
districts, etc) 

Representativeness of sample: 
Measured on All (A), Some (S), 
or None (N) of the units 
representing the intervention 
group in the impact analyses 6 

Component Level Threshold 
for Fidelity of Implementation 
for the Unit that is the Basis 
for the Sample-Level 

Evaluator’s Criteria 
for “Implemented 
with Fidelity” at 
Sample Level 

Component 
Level Fidelity 
Score for the 
Entire Sample 

Implemented 
with Fidelity? 

(Yes, No, N/A) 





High participation at the 
school-level = 

80-100% of teachers with 
high participation 










